Automation agent#1262
Conversation
There was a problem hiding this comment.
main, but PRs should target staged.
The main branch is auto-published from staged and should not receive direct PRs.
Please close this PR and re-open it against the staged branch.
You can change the base branch using the Edit button at the top of this PR,
or run: gh pr edit 1262 --base staged
There was a problem hiding this comment.
Pull request overview
Adds a new custom agent definition intended to guide automated test creation for the RentlyQE framework.
Changes:
- Introduces a new agent file defining workflow, architecture rules, and templates for RentlyQE automation.
- Specifies a large toolset and prescriptive steps for credential handling, DB context retrieval, and TestLink-based test generation.
| tools: ['edit', 'runNotebooks', 'search', 'new', 'runCommands', 'runTasks', 'Copilot Container Tools/*', 'playwright/*', 'tlink-docker-global/*', 'usages', 'vscodeAPI', 'problems', 'changes', 'testFailure', 'openSimpleBrowser', 'fetch', 'githubRepo', 'extensions', 'todos', 'runSubagent'] | ||
| --- | ||
|
|
||
| # RentlyQE Automation Tester Agent |
There was a problem hiding this comment.
docs/README.agents.md doesn’t appear to include this new agent yet. In this repo, agent lists are generated—please run the update script (npm start / npm run build per CONTRIBUTING.md) and commit the resulting docs/README.agents.md changes so the new agent is discoverable.
| @@ -0,0 +1,267 @@ | |||
| --- | |||
There was a problem hiding this comment.
The agent frontmatter is missing name and model. CONTRIBUTING.md’s agent example includes both (and notes required frontmatter fields), and adding them improves listing/filtering on the generated site/README.
| --- | |
| --- | |
| name: 'rently-qe-automation-coding' | |
| model: 'gpt-4.1' |
| [Determine_URL] = Context-aware. Template values: Admin=https://rently.rentlyqeop.com/admin/login, Manager=https://rently.rentlyqeop.com/, Renter=https://homes.rentlyqeop.com/, SmartHome=https://smarthome.rentlyqeop.com. Auto-select when clear, present options if ambiguous. | ||
| [Confirm_Credentials] = REQUIRED. First check existing TestData/{TeamName}/*Data.json for matching credentials. If found: Reuse JSON key. If new: Create entry in TestData/{TeamName}/{PageName}Data.json. Never hardcode. Format: {"loginCredentials":{"username":"test@rently.com","password":"Pass123"}} | ||
| [Retrieve_DB_Context] = MANDATORY. Before code generation, search Weaviate DB (Rently collection) for relevant context. Query: Test case description/feature name. Use semantic_search to retrieve existing implementations, data structures, locators, workflows. Enhances code accuracy and consistency. | ||
| [Reference_TestLink] = DEFAULT SOURCE. Use TestLink as primary test spec source. Tools: mcp_tlink-docker-_search_test_cases_weaviate, mcp_tlink-docker-_list_test_cases, mcp_tlink-docker-_search_testcase_by_id. Workflow: ID/description → Search → Extract specs → Generate code. |
There was a problem hiding this comment.
This agent makes semantic_search mandatory (e.g., for DB context retrieval), but semantic_search is not included in the frontmatter tools list. Either add semantic_search to tools (or configure an MCP server that provides it) or revise the instructions to use only the declared tools.
| [Determine_URL] = Context-aware. Template values: Admin=https://rently.rentlyqeop.com/admin/login, Manager=https://rently.rentlyqeop.com/, Renter=https://homes.rentlyqeop.com/, SmartHome=https://smarthome.rentlyqeop.com. Auto-select when clear, present options if ambiguous. | ||
| [Confirm_Credentials] = REQUIRED. First check existing TestData/{TeamName}/*Data.json for matching credentials. If found: Reuse JSON key. If new: Create entry in TestData/{TeamName}/{PageName}Data.json. Never hardcode. Format: {"loginCredentials":{"username":"test@rently.com","password":"Pass123"}} | ||
| [Retrieve_DB_Context] = MANDATORY. Before code generation, search Weaviate DB (Rently collection) for relevant context. Query: Test case description/feature name. Use semantic_search to retrieve existing implementations, data structures, locators, workflows. Enhances code accuracy and consistency. | ||
| [Reference_TestLink] = DEFAULT SOURCE. Use TestLink as primary test spec source. Tools: mcp_tlink-docker-_search_test_cases_weaviate, mcp_tlink-docker-_list_test_cases, mcp_tlink-docker-_search_testcase_by_id. Workflow: ID/description → Search → Extract specs → Generate code. |
There was a problem hiding this comment.
The TestLink tool names referenced here (mcp_tlink-docker-_...) don’t match the declared tool namespace in frontmatter (tlink-docker-global/*). Please align the documentation with the actual tool IDs the agent can call (and/or add the correct MCP server config) so users aren’t instructed to use non-existent tools.
| [Reference_TestLink] = DEFAULT SOURCE. Use TestLink as primary test spec source. Tools: mcp_tlink-docker-_search_test_cases_weaviate, mcp_tlink-docker-_list_test_cases, mcp_tlink-docker-_search_testcase_by_id. Workflow: ID/description → Search → Extract specs → Generate code. | |
| [Reference_TestLink] = DEFAULT SOURCE. Use TestLink as primary test spec source. Tools: tlink-docker-global/search_test_cases_weaviate, tlink-docker-global/list_test_cases, tlink-docker-global/search_testcase_by_id. Workflow: ID/description → Search → Extract specs → Generate code. |
| @@ -0,0 +1,267 @@ | |||
| --- | |||
| description: 'Automation Agent for RentlyQE Framework' | |||
| tools: ['edit', 'runNotebooks', 'search', 'new', 'runCommands', 'runTasks', 'Copilot Container Tools/*', 'playwright/*', 'tlink-docker-global/*', 'usages', 'vscodeAPI', 'problems', 'changes', 'testFailure', 'openSimpleBrowser', 'fetch', 'githubRepo', 'extensions', 'todos', 'runSubagent'] | |||
There was a problem hiding this comment.
tools includes an entry with spaces (Copilot Container Tools/*). Tool identifiers elsewhere in this repo are slash-delimited IDs without spaces; this may prevent tool resolution. Consider verifying the exact toolset ID and updating it to the canonical name used by Copilot/MCP.
| tools: ['edit', 'runNotebooks', 'search', 'new', 'runCommands', 'runTasks', 'Copilot Container Tools/*', 'playwright/*', 'tlink-docker-global/*', 'usages', 'vscodeAPI', 'problems', 'changes', 'testFailure', 'openSimpleBrowser', 'fetch', 'githubRepo', 'extensions', 'todos', 'runSubagent'] | |
| tools: ['edit', 'runNotebooks', 'search', 'new', 'runCommands', 'runTasks', 'copilot-container-tools/*', 'playwright/*', 'tlink-docker-global/*', 'usages', 'vscodeAPI', 'problems', 'changes', 'testFailure', 'openSimpleBrowser', 'fetch', 'githubRepo', 'extensions', 'todos', 'runSubagent'] |
| [Pre_Development] = Present template OR ask Team Name. Check existing credentials in JSON (reuse if found). Retrieve DB Context from Rently collection (MANDATORY semantic_search with TEST_DESCRIPTION). Determine URL. Confirm/create credentials in JSON. Reference TestLink. Check existing reusable logic. | ||
| [Pre_Development] = Ask Team Name. Retrieve DB Context from Rently collection (MANDATORY semantic_search). Determine URL. Confirm credentials in JSON. Reference TestLink. Check existing reusable logic. |
There was a problem hiding this comment.
The ::QUALITY_CHECKLIST:: has two different [Pre_Development] entries with the same key, which creates ambiguity about which checklist to follow. Recommend merging into a single ordered checklist (or renaming them) to avoid contradictory guidance.
| [Pre_Development] = Present template OR ask Team Name. Check existing credentials in JSON (reuse if found). Retrieve DB Context from Rently collection (MANDATORY semantic_search with TEST_DESCRIPTION). Determine URL. Confirm/create credentials in JSON. Reference TestLink. Check existing reusable logic. | |
| [Pre_Development] = Ask Team Name. Retrieve DB Context from Rently collection (MANDATORY semantic_search). Determine URL. Confirm credentials in JSON. Reference TestLink. Check existing reusable logic. | |
| [Pre_Development] = Present template OR ask Team Name. Retrieve DB Context from Rently collection (MANDATORY semantic_search with TEST_DESCRIPTION). Determine URL. Check existing credentials in JSON (reuse if found) and confirm/create credentials in JSON if missing. Reference TestLink. Check existing reusable logic. |
| TEST_DESCRIPTION = Create new invoice for property | ||
| TESTLINK_ID = TC-BIL-789 | ||
|
|
||
| [Example_No_Login] = |
There was a problem hiding this comment.
[Example_No_Login] is empty and the ::TROUBLESHOOTING:: section begins immediately after it, which makes the examples section hard to follow. Please either provide the no-login example content or remove the placeholder so sections don’t get interleaved.
| [Example_No_Login] = |
| [Missing_Team] = Present template first. If not provided, ask "Which team?" and validate against valid teams | ||
| [Invalid_Template] = If template format invalid, display correct format and request re-submission or switch to Q&A mode | ||
| [Credential_Reuse] = Always check TestData/{Team}/*Data.json before creating new credentials. Reuse existing when possible. | ||
| TESTLINK_ID = TC-RPC-321 |
There was a problem hiding this comment.
There’s a stray TESTLINK_ID = TC-RPC-321 line inside the troubleshooting block, which looks like it belongs in the example templates instead. Moving/removing it would prevent confusing users about where TestLink IDs should be provided.
| TESTLINK_ID = TC-RPC-321 |
Pull Request Checklist
npm startand verified thatREADME.mdis up to date.stagedbranch for this pull request.Description
Type of Contribution
Additional Notes
By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.